The volume, velocity and variety of information available are increasing exponentially, as is the difficulty in assessing its veracity due to the ease of using the Internet as a disinformation vehicle. This requires greater efficiency in the ability to analyze information, including the potential for automated assistance to help human users work through a structured analytical process.
Currently there are a number of solutions for a collaborative analytic technique. They generally fall into two camps: manual and automated. Manual structured analytical techniques—conveyed via written primers or orally through teaching and exposition—typically consist of approaches to collect relevant information, to establish hypotheses and determine causality and to synthesize the data and analysis into a coherent analytical product that aids decision making. But manual techniques are cumbersome and slow. Automated and automation-assisted techniques circumvent the unwieldiness of manual systems and offer the advantage of electronic instantiation of varied analytic approaches as built-in user experience components. But automated analytic techniques are typically cumbersome as well, requiring analytical steps that are not always pertinent to each specific problem. As a result, some systems experience user attrition because electronic instantiations do not effectively manage the trade-off between incorporating a wide array of analytic approaches in the technique and creating a more streamlined user experience.
Furthermore, in both automated and manual structured analytic techniques, the problem of cognitive bias still looms as a constant and undesirable factor. To date, the mitigation of cognitive bias in analytic techniques has only been addressed as an indirect by-product of using the analytical approaches themselves. Developers of analytic techniques have generally taken the position that it is impossible to diminish cognitive biases because of their complexity and universality, and that merely using a collaborative structured analytical technique was the best way to mitigate the critical problem of cognitive bias in analysis. However, no evidence supports this assertion (Pool, R. (Ed.). (2010). Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary. National Academies Press). To the contrary, recent studies have shown that integrating a dedicated cognitive de-biasing training environment into a workflow process can improve prediction accuracy by a statistically significant margin.
In addition to the failure of analytic techniques to mitigate cognitive biases, assessments in conventional analytical techniques are generally registered using imprecise and high-level verbiage rather than quantified scoring. For example, under conventional analytic techniques, users are often asked to rate likelihood or consistency/inconsistency using labels (e.g. low, medium, high, very likely, very unlikely, etc.). Among the reasons given for this are that mathematical precision would be less practical for broad use and not easily understood and that it would be too easy to misuse any simple probability calculation. But recent research on quantitative equivalency of semantic expressions of certainty show analysts can consistently justify a more fine-grained level of precision than words of estimative probability currently allow. (Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., . . . & Murray, T. (2014). Psychological strategies for winning a geopolitical forecasting tournament. Psychological science, 25(5), 1106-1115; Friedman, J. A., & Zeckhauser, R. (2014). Why Assessing Estimative Accuracy is Feasible and Desirable. Intelligence and National Security, 1-23; Friedman, J. A., Baker, J. D., Mellers, B. A., Tetlock, P. E., and Zeckhauser, R. (2015). The Value of Precision in Geopolitical Forecasting: Empirical Foundations for Intelligence Analysis and Foreign Policy Decision Making). It is now generally understood that the elicitation of quantified probabilistic assessments within a large crowd of analysts is not only possible, it is also desirable if performance accountability is to be measured effectively (Barnes, A. (2016). Making intelligence analysis more intelligent: Using numeric probabilities. Intelligence and National Security, 31, No. 3, 327-344; Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., . . . & Murray, T. (2014). Psychological strategies for winning a geopolitical forecasting tournament. Psychological science, 25(5), 1106-1115; Mellers, B., Stone, E., Atanasov, P., Rohrbaugh, N., Metz, S. E., Ungar, L., & Tetlock, P. (2015). The psychology of intelligence analysis: Drivers of prediction accuracy in world politics. Journal of experimental psychology: applied, 21(1), 1; Tetlock, P. E., Mellers, B. A., Rohrbaugh, N., & Chen, E. (2014). Forecasting Tournaments Tools for Increasing Transparency and Improving the Quality of Debate. Current Directions in Psychological Science, 23(4), 290-295). Current analytical techniques that depend upon the use of semantic expressions of likelihood (like “highly likely” or “unlikely”) sacrifice accuracy due to ambiguity in what such semantic expressions actually mean (Friedman, J. A., & Zeckhauser, R. (2012). Assessing Uncertainty in Intelligence′, Intelligence and National Security 27, 4, p. 824-47; Friedman, J. A., & Zeckhauser, R. (2014). Why Assessing Estimative Accuracy is Feasible and Desirable. Intelligence and National Security, 1-23; Friedman, J. A., Baker, J. D., Mellers, B. A., Tetlock, P. E., and Zeckhauser, R. (2015). The Value of Precision in Geopolitical Forecasting: Empirical Foundations for Intelligence Analysis and Foreign Policy Decision Making).
Many conventional structured analytic techniques task users with the dual role of both data imputation (i.e. generating a hypothesis on the basis of data) as well as refutation, despite recent reports that evaluation of hypotheses through dedicated red-teaming can provide a successful method by which analysts can identify weaknesses and challenge assumptions. While crowd-based red-teaming is typically not a feature of conventional structured analytic techniques, studies indicate that red-teaming is best done by a dedicated entity that only conducts falsification, as a way of avoiding confirmation biases that tend to become evident when analysts provide both affirmative reasoning and alternative hypotheses. (Zenko, M. (2015), Red Team: How to Succeed By Thinking Like the Enemy. Council on Foreign Relations, Basic Books, New York).
Finally, collaborative techniques remain slow, inefficient, and difficult to use due to overly complex structures. In computerized processes, one problem stems from leaving a user to choose from a set of differing techniques. A separate issue is that the platform development and distribution model remains tied to desktop-based software. To date, collaborative analytics generally have not provided data structures that enable resource-efficient data handling and options for collaborative analysis while maintaining data consistency across users.